# VoiceAgent Integration with Telephony Networks: Technical Overview
## 1. Voice Streaming over WebSocket
**Sequence Diagram:**
```mermaid
sequenceDiagram
participant AppWeb as App/Web
participant VoiceAgent as VoiceAgent
AppWeb->>VoiceAgent: WebSocket signaling (start/stop/params)
AppWeb-->>VoiceAgent: WebSocket audio stream (continuous push)
VoiceAgent-->>AppWeb: Signaling response (result/status)
```
Signaling and audio streams are both delivered over WebSocket.
**Key Points:**
- Simple and reliable; lightweight architecture, easy to implement and deploy.
- Limited jitter resilience; NAT traversal is reasonable, but performance may degrade on unstable networks.
- Signaling is straightforward and can be RESTful.
**Representative Solutions:**
- [Xiaozhi: An MCP-based Chatbot](https://github.com/78/xiaozhi-esp32)
- [Pipecat: Real-Time Voice & Multimodal AI Agents](https://github.com/pipecat-ai/pipecat)
<div style="page-break-after: always;"></div>
## 2. WebRTC Protocol
**Key Points:**
- Protocol is complex, requiring STUN/TURN servers and multiple network components.
- Native browser support with advanced audio features (noise suppression, echo cancellation).
- Excellent jitter resilience; signaling is flexible and customizable, ideal for internet-based applications.
- **Security Requirement**: Browsers require WebRTC to be used in a Secure Context (**HTTPS** or **127.0.0.1**).
**Sequence Diagram:**
```mermaid
sequenceDiagram
participant AppWeb as App/Web
participant VoiceAgent as VoiceAgent
AppWeb->>VoiceAgent: WebRTC signaling (SDP/ICE/control)
AppWeb-->>VoiceAgent: WebRTC audio stream (SRTP)
VoiceAgent-->>AppWeb: Signaling response (SDP/ICE/status)
```
Signaling is customizable; audio streams are transmitted via WebRTC.
**Representative Solutions:**
- [Ten-Framework: Open-source framework for conversational voice AI agents.](https://github.com/TEN-framework/ten-framework)
- [Pipecat: Real-Time Voice & Multimodal AI Agents](https://github.com/pipecat-ai/pipecat)
- [LiveKit: Real-time video, audio and data for developers](https://github.com/livekit/livekit)
<div style="page-break-after: always;"></div>
## 3. SIP/RTP Protocol
**Key Points:**
- Fully interoperable with legacy telephony systems; standardized and highly compatible.
- Signaling is complex and process-driven, suitable for enterprise and carrier-grade deployments.
- Outstanding jitter resilience.
**Sequence Diagram:**
```mermaid
sequenceDiagram
participant SIPPBX as SIPPBX
participant TelSystem as Telephony System
SIPPBX->>TelSystem: SIP signaling (register/call/hangup)
SIPPBX-->>TelSystem: RTP audio stream (continuous push)
TelSystem-->>SIPPBX: SIP signaling response (status/result)
```
Signaling is handled via SIP; audio streams use RTP.
**Popular SIP PBX Solutions:**
- [FreeSwitch: Software Defined Telecom Stack](https://github.com/signalwire/freeswitch)
FreeSwitch is a widely adopted open-source platform for telephony integration. It powers many enterprise-grade gateways and solutions, offering robust SIP/RTP support and scalability. The project is mature, actively maintained, and features a codebase exceeding 1.8 million lines.
<div style="page-break-after: always;"></div>
## 4. WebRTC Gateway Interoperability Workflow
WebRTC gateways serve as middleware, bridging WebRTC and SIP/RTP networks:
**Sequence Diagram:**
```mermaid
sequenceDiagram
participant AppWeb as App/Web
participant WebRTC Gateway as WebRTC Gateway
participant TelSystem as Telephony System
AppWeb->>WebRTC Gateway: WebRTC signaling (SDP/ICE/control)
AppWeb-->>WebRTC Gateway: WebRTC audio stream (SRTP)
WebRTC Gateway->>TelSystem: SIP signaling (register/call/hangup)
WebRTC Gateway-->>TelSystem: RTP audio stream (continuous push)
TelSystem-->>WebRTC Gateway: SIP signaling response (status/result)
WebRTC Gateway-->>AppWeb: WebRTC signaling response (SDP/ICE/status)
```
1. App/Web clients retain WebRTC advantages (native browser, noise reduction, echo cancellation, jitter resilience).
2. The gateway performs protocol conversion, mapping WebRTC signaling and media streams to SIP/RTP standards for telephony integration.
3. Enables seamless connectivity between internet and telephony networks.
**Key Challenges:**
1. Audio format conversion and resampling are CPU-intensive.
2. High concurrency requirements; traditional thread-based architectures may struggle to scale.
3. Beyond simple forwarding—built-in VAD, noise reduction, ASR, and TTS capabilities simplify application development.
**Representative Solutions:**
- [Janus: the general purpose WebRTC server](https://janus.conf.meetecho.com/)
- [RustPBX: AI-Powered Software-Defined PBX](https://github.com/restsend/rustpbx)
<div style="page-break-after: always;"></div>
## Solution Comparison
| WebSocket | Low | None | Weak | Simple implementation, best for lightweight scenarios |
| WebRTC | Low | None | Strong | Native browser support, advanced audio features |
| SIP/RTP | High | Yes | Strong | Industry standard, highly compatible |
**Recommended Use Cases:**
- WebSocket: LAN, IoT, simple voice dialogs
- WebRTC: Internet voice, smart customer service, conversational AI
- SIP/RTP: Enterprise communications, carrier networks, telephony gateways
For App/Web integration, WebRTC is generally preferred. Interworking with telephony systems requires a gateway for protocol conversion.
<div style="page-break-after: always;"></div>
## About RustPBX
RustPBX was officially launched on February 14, 2025. Prior to this, we developed a SIP protocol stack in Rust (December 4, 2024):
- [rsipstack: A SIP Stack written in Rust](https://github.com/restsend/rsipstack)
RustPBX is designed to be a modern, AI-native replacement for FreeSwitch, with a focus on security and agent-centric features:
- Built-in audio algorithms: VAD, noise reduction, WAV/MP3 playback, recording, CDR
- Native support for ASR/LLM/TTS pipelines, direct cloud integration without MRCP
- Integrated WebRTC and SIP/RTP—no third-party gateway required (effectively Janus + FreeSwitch in one)
- Coroutine-based architecture: a single core can handle 160 concurrent voice channels (VAD + noise reduction + recording)
